The invention relates to image processing, more especially to a method of and apparatus for processing motion picture images taken with a moving camera such as a hand-held camera, or a hand-held terminal device including a camera.
In the future, there is likely to be considerable demand for telephones with a multi-media video and audio capability.
FIG. 1 of the accompanying drawings illustrates one possible design for a video telephone in the form of a hand-held terminal 14. The hand-held terminal has a main housing 10 to which is mounted a video display 12 and an antenna 26. The display is provided for showing moving picture images received by the terminal from a wireless transmission to the antenna 26. A camera 16 and 18 for taking images is built into the housing 10. The camera is provided to take a sequence of image frames and to supply them to the antenna 26 for wireless transmission to a base station. The camera will most likely be a digital camera based on a charged coupled device (CCD) 16, or other array detector, and will have conventional lens optics 18, possibly in conjunction with optical fibre components. The camera will have an optical axis xe2x80x9cOxe2x80x9d. In addition, there will be a notional vertical axis xe2x80x9cVxe2x80x9d of the terminal defined by vertical alignment of the display and camera. The alignment of the display 12 will be made to coincide with the alignment of the projection of the image viewed by the camera on the rectangular active area of the CCD chip 16. The antenna 26 may be a broad-band transceiver antenna 26, or some other antenna arrangement such as separate helical antennae for receiving and transmitting arranged within the housing 10. The main housing 10 will also comprise various keys or buttons for dialling and other functions, and have an in-built loudspeaker and microphone for the audio part of the signal. These components are not shown.
Consider a video telephone communication between two users, Janet and John. During the call, Janet will hold her terminal by one or two hands for comfortable viewing of John on the display. For Janet, whether or not she is holding her terminal at the correct orientation will be of secondary importance. However, for John, any mis-orientation of Janet""s terminal will be a problem, since it will result in Janet""s image being mis-oriented on the display of John""s hand-held terminal. For John, this will be a nuisance and detract from his subjective evaluation of picture quality.
FIGS. 2 to 4 of the accompanying drawings illustrate the orientation problem.
FIG. 2 of the accompanying drawings illustrates Janet""s image displayed on John""s hand-held terminal with proper alignment of Janet""s hand-held terminal relative to herself. The image is shown as a number of shaded objects, as would result from use of a standard such as MPEG-4. Janet is object 3, the remaining objects 1, 2 and 4 being background objects.
FIG. 3 of the accompanying drawings shows Janet""s image as superimposed on the CCD chip 16 of her hand-held terminal, which is now being held by her tilted at an angle. More particularly, the vertical axis xe2x80x9cVxe2x80x9d of the hand-held terminal now extends at an angle xcex8 to an axis xe2x80x9cUxe2x80x9d characteristic of Janet""s image.
FIG. 4 of the accompanying drawings shows Janet""s image as it appears on the display of John""s hand-held terminal when Janet is holding her terminal as shown in FIG. 3. Thus, if John holds his terminal straight, Janet appears to be leaning over. John could re-align Janet""s image by rotating his terminal, but this would affect his image as displayed on Janet""s terminal. Reaction times and transmission lag could result in an unstable picture orientation if communicating parties attempt hand correction of the vertical alignment in this way.
More generally, in any application where there is a possibility of rotating a camera about its optical axis, the image taken by the camera will appear distorted in the perception of a viewer when displayed on a remote terminal. For example, a door or building will appear to be leaning over at an angle, or the horizon of a landscape will appear tilted.
Appreciation of this problem leads to the following conclusions for hand-held terminals comprising a display and a built-in video camera:
(i) It will be inconvenient and difficult for a user to hold a hand-held terminal so that his/her image aligns vertically with a vertical axis defined by the hand-held terminal""s camera and display.
(ii) The vast majority of images of interest will have a preferred vertical alignment axis that will need to be aligned with a vertical axis of the transmitting user""s hand-held terminal for maximum perceived picture quality on the receiving user""s display.
It is therefore an object of the invention to provide a method and apparatus by which vertical mis-alignment of images taken with a hand-held camera device can be automatically corrected for.
According to a first aspect of the invention there is provided a hand-held device comprising a display for displaying moving pictures on a frame-by-frame basis and a camera having an optical axis extending generally away from the display to image a person who is viewing the display. The hand-held device further comprises a sensor configured to determine a rotational angle between an alignment axis of the hand-held device and a reference alignment axis in real space, and a signal processing circuit arranged to associate image frames taken by the camera with respective rotational angles determined by the sensor.
By associating each frame with a rotational angle reflecting vertical mis-alignment of the data content of the image, vertical mis-alignment can be corrected for by applying a rotational transform to the image frames, either in the hand-held device itself or subsequently.
In one embodiment, a digital signal processor is operatively arranged between the camera""s detector and an output stage of the hand-held device so as to apply a rotational transform to each image frame taken by the camera prior to supply of that frame to the output stage.
In another embodiment, a digital signal processor is operatively arranged between an input stage of a terminal device and its display so as to apply a rotational transform to each image frame received by the input stage prior to supply to the display, the transform being a rotation of the image frame through an angle derived from the rotational angle associated with that frame which is supplied to the hand-held device with the image data device. In this embodiment, the terminal device may in fact not be a hand-held device, but could be a larger device such as a bulky projector, home video player or personal computer.
In a further embodiment, a digital signal processor for applying the rotational transform is arranged in a wireless base station used for relaying data between transceiver parties. The transform angle is derived from the rotational angle associated with that frame which is supplied to the base station with the image data by the transmitting party.
According to a second aspect of the invention there is provided an image processing apparatus, comprising a digital signal processor for processing a sequence of image frames by: (a) determining a vertical alignment axis for each frame of the sequence from an analysis of the data content of that frame; (b) applying a rotational transform to each frame to map the vertical alignment axis determined by the analysis onto a fixed alignment axis of the frame; and (c) outputting the sequence of image frames. This approach differs from that of the first aspect of the invention in that the vertical mis-alignment is determined from image processing of the data content of the image frames themselves, rather than by an independent measurement of a physical parameter, such as gravity, with a sensor.
In one embodiment, the image processing apparatus of the second aspect of the invention is provided in a hand-held device comprising a camera, the image processing apparatus being connected on an output side of the camera to apply rotational transformations to frames obtained by the camera, thereby to compensate for vertical misalignment of the data content of the frames.
In another embodiment, the image processing apparatus of the second aspect of the invention is provided in a video display device, the image processing apparatus being connected in the data path leading to the display, thereby to compensate for vertical misalignment of the data content of the frames supplied for display. The display device may be a personal computer, a hand-held video telephone, or a micro-mirror projector, for example.
In a further embodiment, a base station for wireless communication between a plurality of transceiver devices is equipped with an image processing apparatus according to the second aspect of the invention to compensate for vertical misalignment of the data content of the image frames received by the base station from a transmitting party prior to relaying the signal to a receiving party.
Further aspects of the invention are exemplified by the attached claims.